8. Least squares. ˆ Review of linear equations. ˆ Least squares. ˆ Example: curve-fitting. ˆ Vector norms. ˆ Geometrical intuition

Transcription

1 CS/ECE/ISyE 54 Introduction to Optimization Spring Least squares ˆ Review of linear equations ˆ Least squares ˆ Eample: curve-fitting ˆ Vector norms ˆ Geometrical intuition Laurent Lessard (

2 Review of linear equations System of m linear equations in n unknowns: a a 1n n = b 1 a a n n = b... a m a mn n = b m a a 1n 1 b =. a m1... a mn n b m Compact representation: A = b. Only three possibilities: 1. eactly one solution (e.g. 1 + = 3 and 1 = 1). infinitely many solutions (e.g. 1 + = 0) 3. no solutions (e.g. 1 + = 1 and 1 + = ) 8-

3 Review of linear equations ˆ column interpretation: the vector b is a linear combination of {a 1,..., a n }, the columns of A. 1 A = [ ] a 1 a... a n. = a a n n = b n The solution tells us how the vectors a i can be combined in order to produce b. ˆ can be visualized in the output space R m. 8-3

4 Review of linear equations ˆ row interpretation: the intersection of hyperplanes ãi T = b i where ãi T is the i th row of A. ã1 T ã ã T 1 T b 1 A =. = ã T. = b. ãm T ã T m The solution is a point at the intersection of the affine hyperplanes. Each ã i is a normal vector to a hyperplane. ˆ can be visualized in the input space R n. b m 8-4

5 Review of linear equations ˆ The set of solutions of A = b is an affine subspace. ˆ If m > n, there is (usually but not always) no solution. This is the case where A is tall (overdetermined). Can we find so that A b? One possibility is to use least squares. ˆ If m < n, there are infinitely many solutions. This is the case where A is wide (underdetermined). Among all solutions to A = b, which one should we pick? One possibility is to use regularization. In this lecture, we will discuss least squares. 8-5

6 Least squares ˆ Typical case of interest: m > n (overdetermined). If there is no solution to A = b we try instead to have A b. ˆ The least-squares approach: make Euclidean norm A b as small as possible. ˆ Equivalently: make A b as small as possible. Standard form: minimize A b It s an unconstrained optimization problem. 8-6

7 Least squares ˆ Typical case of interest: m > n (overdetermined). If there is no solution to A = b we try instead to have A b. ˆ The least-squares approach: make Euclidean norm A b as small as possible. ˆ Equivalently: make A b as small as possible. Properties: ˆ = n = T ˆ In Julia: = norm() ˆ In JuMP: = dot(,) = sum(.^) 8-7

8 Least squares ˆ column interpretation: find the linear combination of columns {a 1,..., a n } that is closest to b. A b = (a a n n ) b b a 1 a a a 8-8

9 Least squares ˆ row interpretation: If ãi T is the i th row of A, define r i := ãi T b i to be the i th residual component. A b = (ã T 1 b 1 ) + + (ã T m b m ) We minimize the sum of squares of the residuals. ˆ Solving A = b would make all residual components zero. Least squares attempts to make all of them small. 8-9

10 Eample: curve-fitting ˆ We are given noisy data points ( i, y i ). ˆ We suspect they are related by y = p + q + r ˆ Find the p, q, r that best agrees with the data. Writing all the equations: y 1 p1 + q 1 + r y p + q + r =. y m pm + q m + r y y. 1 p q... r m m 1 y m ˆ Also called regression 8-10

11 Eample: curve-fitting ˆ More complicated: y = pe + q cos() r + s 3 ˆ Find the p, q, r, s that best agrees with the data. Writing all the equations: y 1 e 1 cos( 1 ) y. e cos( ) p 3 q.... y m e m cos( m ) r m m 3 s ˆ Julia notebook: Regression.ipynb 8-11

12 Vector norms We want to solve A = b, but there is no solution. Define the residual to be the quantity r := b A. We can t make it zero, so instead we try to make it small. Many options! ˆ minimize the largest component (a.k.a. the -norm) r = ma r i i ˆ minimize the sum of absolute values (a.k.a. the 1-norm) r 1 = r 1 + r + + r m ˆ minimize the Euclidean norm (a.k.a. the -norm) r = r = r1 + r + + r m 8-1

13 Vector norms Eample: find [ ] that is closest to [ ] 1. Blue line is the set of points with coordinates (, ). y 4 3 Find the one closest to the red point located at (1, ). Answer depends on your notion of distance!

14 Vector norms Eample: find [ ] that is closest to [ ] 1. f() 4 Minimize largest component: min ma{ 1, } Optimum is at =

15 Vector norms Eample: find [ ] that is closest to [ ] 1. f() 4 Minimize sum of components: min 1 + Optimum is any

16 Vector norms Eample: find [ ] that is closest to [ ] 1. f() 4 Minimize sum of squares: min ( 1) + ( ) Optimum is at =

17 Vector norms Eample: find [ ] that is closest to [ ] 1. Equivalently, we can: Minimize sum of squares min ( 1) + ( ) Optimum is at = 1.5. f()

18 Vector norms ˆ minimizing the largest component is an LP: min ma ãt i r i min,t i s.t. ˆ minimizing the sum of absolute values is an LP: min m ãt i r i min i=1,t i s.t. ˆ minimizing the -norm is not an LP! m (ãt ) min i r i i=1 t t ã T i r i t t t m t i ã T i r i t i 8-18

19 Geometry of LS b b Aˆ Aˆ a a 1 ˆ The set of points {A} is a subspace. ˆ We want to find ˆ such that Aˆ is closest to b. ˆ Insight: (b Aˆ) must be orthogonal to all line segments contained in the subspace. 8-19

20 Geometry of LS b b Aˆ Aˆ a a 1 ˆ Must have: (Aˆ Az) T (b Aˆ) = 0 for all z ˆ Simplifies to: (ˆ z) T (A T b A T Aˆ) = 0. Since this holds for all z, the normal equations are satisfied: A T A ˆ = A T b 8-0

21 Normal equations Theorem: If ˆ satisfies the normal equations, then ˆ is a solution to the least-squares optimization problem minimize A b Proof: Suppose A T A ˆ = A T b. Let be any other point. A b = A( ˆ) + (Aˆ b) = A( ˆ) + Aˆ b + ( ˆ) T A T (Aˆ b) = A( ˆ) + Aˆ b Aˆ b 8-1

22 Normal equations Least squares problems are easy to solve! ˆ Solving a least squares problem amounts to solving the normal equations. ˆ Normal equations can be solved in a variety of standard ways: LU (Cholesky) factorization, for eample. ˆ More specialized methods are available if A is very large, sparse, or has a particular structure that can be eploited. ˆ Comparable to LPs in terms of solution difficulty. 8-

23 Least squares in Julia 1. Using JuMP: using JuMP, Gurobi m = m, [1:size(A,)] m, Min, sum((a*-b).^) ) solve(m) Note: only Gurobi or Mosek currently support this synta. Solving the normal equations directly: = inv(a *A)*(A *b) Note: Requires A to have full column rank (A T A invertible) 3. Using the backslash operator (similar to Matlab): = A\b Note: Fastest and most reliable option! 8-3